Comparison of Sampling Techniques on the Performance of Monte- Carlo Based Sensitivity Analysis

Sensitivity analysis is a key part of a comprehensive energy simulation study. Monte-Carlo techniques have been successfully applied to many simulation tools. Several sampling techniques have been proposed in the literature; however to date there has been no comparison of their performance for typical building simulation applications. This paper examines the performance of simple random, stratified and Latin Hypercube sampling when applied to a typical building simulation problem. An integrated natural ventilation problem was selected as it has an inexpensive calculation time thus allowing multiple sensitivity analyses to be undertaken, while being realistic as wind and temperature effects are both modeled. The research shows that compared to simple random sampling: LHS and stratified sampling produce results that are not significantly different (at a 5% level) with increased robustness (less variance in the mean prediction). However, it should not be inferred from this that fewer simulation runs are required for LHS and stratified sampling. Given the results presented here and in previous work it would indicate that for practical purposes Monte-Carlo uncertainty analysis in typical building simulation applications should use about 100 runs and simple random sampling. INTRODUCTION The field of sensitivity analysis is becoming more commonplace in building simulation. Early work by Lomas and Eppel (1992) compared the performance of three techniques (differential, Monte-Carlo and stochastic sensitivity analysis). Following on from this Macdonald (2002) embedded sensitivity analysis in ESP-r and de Wit (2001) applied the Monte-Carlo technique to the analysis of natural ventilation and thermal comfort. Since then several authors have used the Monte-Carlo technique on a diverse range of building simulation applications (for example: Hyun et al 2007 and Kotek et al 2007). To date the effect of sampling technique on the results has not been analysed in the above publications. It is known that stratified sampling can introduce an unknown bias into the results of the analysis (discussed briefly by Macdonald (2002) and de Wit (2001)) and that there can be varying degrees of success with Latin Hypercube (Saltelli et al 2000). This paper will examine three sampling techniques (simple random, stratified and Latin hypercube) and compare their performance. Sampling techniques It is standard statistical procedure to use sampling techniques to improve the coverage of the sample, especially when the function being analysed is expensive. The aim of a sampling strategy is to reduce the variance in the estimate of the mean. When applied to building simulation this is attractive as the evaluation of the simulation results (e.g. building energy consumption) can be costly, therefore any reduction in the number of simulations required for a Monte-Carlo analysis will result in a reduction in computational effort. Three sampling techniques will be described: simple random sampling. Stratified sampling and Latin Hypercube sampling (LHS). Simple random sampling This is the most basic sampling technique described here and will be used as the basis for comparisons. The method works by generating a random number and scaling this to the target variable via its probability distribution. The method conforms to the laws of statistics. The mean of the estimate is: